10 research outputs found
Counting to Ten with Two Fingers: Compressed Counting with Spiking Neurons
We consider the task of measuring time with probabilistic threshold gates implemented by bio-inspired spiking neurons. In the model of spiking neural networks, network evolves in discrete rounds, where in each round, neurons fire in pulses in response to a sufficiently high membrane potential. This potential is induced by spikes from neighboring neurons that fired in the previous round, which can have either an excitatory or inhibitory effect.
Discovering the underlying mechanisms by which the brain perceives the duration of time is one of the largest open enigma in computational neuro-science. To gain a better algorithmic understanding onto these processes, we introduce the neural timer problem. In this problem, one is given a time parameter t, an input neuron x, and an output neuron y. It is then required to design a minimum sized neural network (measured by the number of auxiliary neurons) in which every spike from x in a given round i, makes the output y fire for the subsequent t consecutive rounds.
We first consider a deterministic implementation of a neural timer and show that Theta(log t) (deterministic) threshold gates are both sufficient and necessary. This raised the question of whether randomness can be leveraged to reduce the number of neurons. We answer this question in the affirmative by considering neural timers with spiking neurons where the neuron y is required to fire for t consecutive rounds with probability at least 1-delta, and should stop firing after at most 2t rounds with probability 1-delta for some input parameter delta in (0,1). Our key result is a construction of a neural timer with O(log log 1/delta) spiking neurons. Interestingly, this construction uses only one spiking neuron, while the remaining neurons can be deterministic threshold gates. We complement this construction with a matching lower bound of Omega(min{log log 1/delta, log t}) neurons. This provides the first separation between deterministic and randomized constructions in the setting of spiking neural networks.
Finally, we demonstrate the usefulness of compressed counting networks for synchronizing neural networks. In the spirit of distributed synchronizers [Awerbuch-Peleg, FOCS\u2790], we provide a general transformation (or simulation) that can take any synchronized network solution and simulate it in an asynchronous setting (where edges have arbitrary response latencies) while incurring a small overhead w.r.t the number of neurons and computation time
The Computational Cost of Asynchronous Neural Communication
Biological neural computation is inherently asynchronous due to large variations in neuronal spike timing and transmission delays. So-far, most theoretical work on neural networks assumes the synchronous setting where neurons fire simultaneously in discrete rounds. In this work we aim at understanding the barriers of asynchronous neural computation from an algorithmic perspective. We consider an extension of the widely studied model of synchronized spiking neurons [Maass, Neural Networks 97] to the asynchronous setting by taking into account edge and node delays.
- Edge Delays: We define an asynchronous model for spiking neurons in which the latency values (i.e., transmission delays) of non self-loop edges vary adversarially over time. This extends the recent work of [Hitron and Parter, ESA\u2719] in which the latency values are restricted to be fixed over time. Our first contribution is an impossibility result that implies that the assumption that self-loop edges have no delays (as assumed in Hitron and Parter) is indeed necessary. Interestingly, in real biological networks self-loop edges (a.k.a. autapse) are indeed free of delays, and the latter has been noted by neuroscientists to be crucial for network synchronization.
To capture the computational challenges in this setting, we first consider the implementation of a single NOT gate. This simple function already captures the fundamental difficulties in the asynchronous setting. Our key technical results are space and time upper and lower bounds for the NOT function, our time bounds are tight. In the spirit of the distributed synchronizers [Awerbuch and Peleg, FOCS\u2790] and following [Hitron and Parter, ESA\u2719], we then provide a general synchronizer machinery. Our construction is very modular and it is based on efficient circuit implementation of threshold gates. The complexity of our scheme is measured by the overhead in the number of neurons and the computation time, both are shown to be polynomial in the largest latency value, and the largest incoming degree ? of the original network.
- Node Delays: We introduce the study of asynchronous communication due to variations in the response rates of the neurons in the network. In real brain networks, the round duration varies between different neurons in the network. Our key result is a simulation methodology that allows one to transform the above mentioned synchronized solution under edge delays into a synchronized under node delays while incurring a small overhead w.r.t space and time
Random Sketching, Clustering, and Short-Term Memory in Spiking Neural Networks
We study input compression in a biologically inspired model of neural computation. We demonstrate that a network consisting of a random projection step (implemented via random synaptic connectivity) followed by a sparsification step (implemented via winner-take-all competition) can reduce well-separated high-dimensional input vectors to well-separated low-dimensional vectors. By augmenting our network with a third module, we can efficiently map each input (along with any small perturbations of the input) to a unique representative neuron, solving a neural clustering problem.
Both the size of our network and its processing time, i.e., the time it takes the network to compute the compressed output given a presented input, are independent of the (potentially large) dimension of the input patterns and depend only on the number of distinct inputs that the network must encode and the pairwise relative Hamming distance between these inputs. The first two steps of our construction mirror known biological networks, for example, in the fruit fly olfactory system [Caron et al., 2013; Lin et al., 2014; Dasgupta et al., 2017]. Our analysis helps provide a theoretical understanding of these networks and lay a foundation for how random compression and input memorization may be implemented in biological neural networks.
Technically, a contribution in our network design is the implementation of a short-term memory. Our network can be given a desired memory time t_m as an input parameter and satisfies the following with high probability: any pattern presented several times within a time window of t_m rounds will be mapped to a single representative output neuron. However, a pattern not presented for c?t_m rounds for some constant c>1 will be "forgotten", and its representative output neuron will be released, to accommodate newly introduced patterns
Broadcast CONGEST Algorithms against Adversarial Edges
We consider the corner-stone broadcast task with an adaptive adversary that
controls a fixed number of edges in the input communication graph. In this
model, the adversary sees the entire communication in the network and the
random coins of the nodes, while maliciously manipulating the messages sent
through a set of edges (unknown to the nodes). Since the influential work
of [Pease, Shostak and Lamport, JACM'80], broadcast algorithms against
plentiful adversarial models have been studied in both theory and practice for
over more than four decades. Despite this extensive research, there is no round
efficient broadcast algorithm for general graphs in the CONGEST model of
distributed computing. We provide the first round-efficient broadcast
algorithms against adaptive edge adversaries. Our two key results for -node
graphs of diameter are as follows:
1. For , there is a deterministic algorithm that solves the problem
within rounds, provided that the graph is 3
edge-connected. This round complexity beats the natural barrier of
rounds, the existential lower bound on the maximal length of edge-disjoint
paths between a given pair of nodes in . This algorithm can be extended to a
-round algorithm against adversarial edges in
edge-connected graphs.
2. For expander graphs with minimum degree of , there is
an improved broadcast algorithm with rounds against
adversarial edges. This algorithm exploits the connectivity and conductance
properties of G-subgraphs obtained by employing the Karger's edge sampling
technique.
Our algorithms mark a new connection between the areas of fault-tolerant
network design and reliable distributed communication.Comment: accepted to DISC2
Secure Distributed Network Optimization Against Eavesdroppers
We present a new algorithmic framework for distributed network optimization in the presence of eavesdropper adversaries, also known as passive wiretappers. In this setting, the adversary is listening to the traffic exchanged over a fixed set of edges in the graph, trying to extract information on the private input and output of the vertices. A distributed algorithm is denoted as f-secure, if it guarantees that the adversary learns nothing on the input and output for the vertices, provided that it controls at most f graph edges.
Recent work has presented general simulation results for f-secure algorithms, with a round overhead of D^?(f), where D is the diameter of the graph. In this paper, we present a completely different white-box, and yet quite general, approach for obtaining f-secure algorithms for fundamental network optimization tasks. Specifically, for n-vertex D-diameter graphs with (unweighted) edge-connectivity ?(f), there are f-secure congest algorithms for computing MST, partwise aggregation, and (1+?) (weighted) minimum cut approximation, within O?(D+f ?n) congest rounds, hence nearly tight for f = O?(1).
Our algorithms are based on designing a secure algorithmic-toolkit that leverages the special structure of congest algorithms for global optimization graph problems. One of these tools is a general secure compiler that simulates light-weight distributed algorithms in a congestion-sensitive manner. We believe that these tools set the ground for designing additional secure solutions in the congest model and beyond
Spiking Neural Networks Through the Lens of Streaming Algorithms
We initiate the study of biological neural networks from the perspective of
streaming algorithms. Like computers, human brains suffer from memory
limitations which pose a significant obstacle when processing large scale and
dynamically changing data. In computer science, these challenges are captured
by the well-known streaming model, which can be traced back to Munro and
Paterson `78 and has had significant impact in theory and beyond. In the
classical streaming setting, one must compute some function of a stream of
updates , given restricted single-pass access
to the stream. The primary complexity measure is the space used by the
algorithm.
We take the first steps towards understanding the connection between
streaming and neural algorithms. On the upper bound side, we design neural
algorithms based on known streaming algorithms for fundamental tasks, including
distinct elements, approximate median, heavy hitters, and more. The number of
neurons in our neural solutions almost matches the space bounds of the
corresponding streaming algorithms. As a general algorithmic primitive, we show
how to implement the important streaming technique of linear sketching
efficient in spiking neural networks. On the lower bound side, we give a
generic reduction, showing that any space-efficient spiking neural network can
be simulated by a space-efficiently streaming algorithm. This reduction lets us
translate streaming-space lower bounds into nearly matching neural-space lower
bounds, establishing a close connection between these two models.Comment: To appear in DISC'20, shorten abstrac